1. Field of the Invention
This invention relates to an image shake detecting device which is highly suited for the automatic image stabilizing device of a video camera or the like and also to a device for controlling the image stabilizing device.
2. Description of the Related Art
In the image sensing optical apparatuses of varied kinds including video cameras, etc., irrespective as to whether they are adapted for industrial instrumentation, or consumer appliances, shaking of an image not only hinders easy image sighting but also degrades image recognizing accuracy. In the case of the video camera, for example, the camera is often operated while the operator is walking or while the camera is on a moving vehicle. In such a case, it is inevitable to have a sensed image shaken by a shake of the camera according to the photographing conditions or the object to be photographed.
To solve this problem, there have been proposed image shake detecting devices adopting varied methods. In one of such methods, the movement of the camera is physically detected by means of an acceleration sensor (an angular velocity sensor) and an optical system is compensated for the movement according to the direction and the degree of the movement. In another conceivable method, the parallel moving extent of the whole image plane is detected through a video signal and is expressed in a movement vector. Then, the optical system is compensated on the basis of the vector.
In accordance with the method of using the acceleration sensor, the size of the device becomes larger thus requiring increases in space and weight. Besides, it results in a complex structural arrangement. This method is therefore hardly suited for a home video camera such as a camera-incorporating type video tape recorder which must be compact in size and light in weight.
As regards the method of computing and obtaining the movement vector of the image plane from the video signal, some camera movement that is intentionally caused by the operator might be mistakenly detected for a shake of the image. The device also would respond to a movement of the object which is in realty not a shake of the image. That method thus also has a serious problem.
The above-stated known image shake detecting devices include, for example, an image stabilizing camera which is disclosed in Japanese Laid-Open Patent Application No. SHO 61-248681. The camera of this kind is arranged as follows: an optical image is converted into an electrical signal by means of an image sensing system which consists of a lens system and a photo-electric converting element. A TV image signal is obtained through a signal processing operation performed in a given manner on the electrical signal by a signal processing circuit. The image signal thus obtained is supplied to an image shake detecting circuit as well as to a monitor. A correlation between two image planes obtained at a given time interval is detected from the image signal to find the degree and the direction of any image shake. Then, a driving control circuit and a motor are operated-to control and move the lens system to offset the image shaking on the basis of the result of detection. The camera is thus arranged to be capable of obtaining a stable image even when the camera shakes.
However, the image shake detecting device which is arranged in this manner is incapable of making a discrimination between the movement of an object occurring only in a part of the image plane and a shake of the whole image plane. To solve this problem, the image shake detecting sensitivity of the device must be arranged to vary for the different areas of the image plane.
In connection with this problem, an image shake detecting device has been proposed as disclosed in an article entitled xe2x80x9cAbout an Image Plane Shake Compensating Device,xe2x80x9d appeared in xe2x80x9cThe Technical Reportxe2x80x9d of the Television Society, Vol. 11, No. 3, p 43 to 48, PPOE, ""87-12 (May, 1987). In the case of that device, the whole image plane is divided into 140 blocks of areas. A shake detection switch is arbitrarily turned on or off for each of these areas and an image shake is detected only from the areas for which the shake detection switch is turned on in accordance with a representing point matching method.
In accordance with the arrangement of this image shake detecting device, however, an image to be used as reference must be temporarily stored at a frame memory with its varied density values kept intact. To meet this requirement, the device necessitates the use of an analog-to-digital (hereinafter referred to as A/D) converter and a memory of a relatively large capacity. In addition to this shortcoming, the device is arranged to have one image plane superposed on another by staggering them to a certain degree of vector and to find out a vector that gives the highest degree of coincidence. Therefore, the operation of the device includes a large amount of computation. The device thus necessitates a circuit arrangement on a large scale and requires a long computing time.
Besides, it has been extremely difficult to have the image shake detecting device incorporated in a compact video camera which must be capable of carrying out real-time processing and must be arranged in a compact circuit arrangement, because: The above-stated device of the prior art necessitates the use of a large circuit arrangement including the A/D converter, the frame memory, a computing circuit, etc. and a long processing time.
Further, a control system which is provided for actually effecting shake compensation by driving a photo-taking optical system on the basis of information on a shake detected by the above-stated image shake detecting devices has been arranged as follows: Various methods have been known for correcting a displacement of an image. In one of these varied methods which has recently been contrived, there are provided optical compensation members such as a lens which is arranged to have its optical axis tiltable by a motor and a variable apex-angle prism. With an image obtained through an optical system which includes these optical compensation members, the deflected degree and the size of the image to be compensated on the basis of the image displacement are detected from the image. Then, the so-called feedback control is performed to deflect the image by driving the above-stated optical compensation members in accordance with information on the results of detection.
In the case of the device of this kind, a variable apex-angle prism which is arranged to have its apex angle variable is employed as an optical image deflecting means. A motor is used for driving the prism. The deflected degree of the image is detected in the following manner: The images of at least two TV camera image planes having a time difference between them are compared with each other. The deflection of the variable apex-angle prism is then controlled in such a way as to lessen displacement occurring between the two images compared, so that the images can be obtained always in a coinciding state. Further, in cases where the object to be photographed is traced (tracked) or where the object moves, the variable apex-angle prism is driven in such a way as to lessen the degree of the positional change of the object""s image within the image plane. In a case where the image is shaken by a shake of the camera caused by something, the image is stabilized by driving the variable apex-angle prism in the same way as in the case of tracing the object to lessen the degree of the positional change of the object""s image. Therefore, in these cases, the control is thus performed virtually in the same manner.
However, an interlaced scanning method is used for the TV camera or the like in general. In the case of the TV camera of the NTSC system, one image plane is formed and transmitted for every period of {fraction (1/60)} sec (one field period). In a case where the feedback control method is employed for detecting the state of the image obtained by such a TV camera, the data of the feedback system consists of samples dispersed at intervals of at least {fraction (1/60)} sec. Besides, the feedback action delays by a period of time required for a detection process plus a period of time required for image plane transmission (about {fraction (1/60)} sec). The control output of the device, therefore, is unstable for any quick movement of the object. The response characteristic of the device is poor. In a case where the motor used for driving the optical compensation member has a good rise characteristic, a feedback coefficient exceeding 1 tends to cause oscillation. A further drawback of the prior art device resides in a poor frequency characteristic. These problems stem from the fact that the control is performed by using the control algorithm of a continuous time system while sampling by detecting means has a dispersive time delay.
Further, there has been proposed an apparatus which uses an image shake detecting device for objects appearing within an image plane and is arranged to trace (track) a moving object and to continuously perform control for accurate automatic focusing and accurate automatic exposure control. For example, apparatuses of this kind have been disclosed in U.S. patent applications Ser. No. 737,163 filed on May 23, 1985; Ser. No. 106,427 filed on Oct. 8, 1987; Ser. No. 154,078 filed on Feb. 9, 1988; Ser. No. 237,511 filed on Aug. 26, 1988; Ser. No. 240,915 filed on Sep. 6, 1988; Ser. No. 258,692 filed on Oct. 17, 1988; and Ser. No. 264,204 filed on Oct. 28, 1988. However, as mentioned in the foregoing, it is difficult to accurately make a discrimination between the movement of only one object in a part of the image plane and the movement of the whole image plane due to the movement of the camera. In accordance with the above-stated method disclosed in Japanese Laid-Open Patent Application No. SHO 61-248681, the degree of accuracy would lower if there is no luminance difference because the method utilizes a difference in luminance between the background and the object to be photographed. Further, in accordance with the technique disclosed in the above-stated Technical Report of the Television Society, it becomes difficult to accurately determine the areas in a case where a plurality of objects come to move. In other words, each of the methods of the prior art has a disadvantage as well as its advantage. Therefore, with the prior art image shake detecting devices incorporated in a compact video camera in accordance with these methods, they are incapable of adequately coping with every image conditions.
With respect to the image shake detecting device of this kind, further examples have been disclosed in U.S. patent application Ser. No. 855,732 filed on Apr. 25, 1986, U.S. patent application Ser. No. 880,152 filed on Jun. 30, 1986, etc. These patent applications disclose an image stabilizing camera which is arranged to compensate the optical axis of the lens system by detecting an image shake through the edge component of the image of an object to be photographed; and an arrangement to recognize an image through a histogram which represents the distribution of the feature of the image.
This invention is directed to the solution of the above-stated problems of the prior art. It is a first object of the invention to provide an image shake detecting device which is capable of accurately and stably detecting shaking of an image plane without mistaking a movement of an image resulting from shaking of the camera for an image movement resulting from a movement of a photographing object or from a panning movement of the camera or the like.
It is another object of the invention to provide an image shake detecting device for a camera which is arranged to detect a shaking state of either the image of a main object to be photographed or that of the background, whichever suits better to the state of the object, so that shake compensation can be most appositely made under all conditions including such cases where a picture of a moving object is to be taken with the camera fixed or with the camera moved to trace (track) the moving object.
It is a further object of the invention to provide an image shake detecting device which is simple in structural arrangement and excels in responsivity without necessitating the use of any special sensor.
To attain this object, an image shake detecting device which is arranged as a preferred embodiment of this invention to detect a shake of an image on an image sensing plane on the basis of an image signal output from image sensing means comprises: first detection means for detecting the displacement of the image on the basis of an image signal corresponding to a first detection area set on the image sensing plane; second detection means for detecting the displacement of the image on the basis of an image signal corresponding to a second detection area set on the image sensing plane; switching means for selecting one of the first and second detection means; and image shake detecting means for computing the shake of the image on the basis of an output of the first or second detection means selected by the switching means.
It is a further object of the invention to provide an image shake detecting and compensating device which performs no faulty action under any image plane condition.
It is a further object of the invention to provide an image shake detecting device which performs no faulty action even in cases where a feature of a main object to be photographed and that of the background of the object are hardly distinguishable from each other due to the low level of a high frequency component or sharpness, etc. obtained within the image plane.
To attain that object, an image shake detecting device which is arranged as a preferred embodiment of the invention to detect a shake of an image on an image sensing plane on the basis of an image signal output from image sensing means comprises: movement detecting means for detecting the displacement of the image on the basis of an image signal corresponding to a predetermined detection area set on the image sensing plane; compensating means for compensating for a movement of the image, by displacing an optical axis of a photo-taking optical system which images an incident light on an imaging plane of the image sensing means, on the basis of information on the displacement of the image output from the movement detecting means; sharpness detecting means for detecting a signal component indicative of a sharpness of the image on the basis of the image signal corresponding to the detection area; and control means for controlling an action of the compensating means on the basis of an output of the sharpness detecting means.
It is a further object of the invention to provide an image shake detecting device which is arranged to measure any deviation that occurs at a given feature point within each image plane during the process of scanning image planes which differ timewise from each other and to detect a quantity of the shake of an image on the basis of a measured value thus obtained.
It is a further object of the invention to provide an image shake detecting and compensating device which has the detection sensitivity for a quantity of the shake arranged to be variable and is capable of accurately discriminating a shake of the whole image plane from a local shake occurring on the image plane, so that the device always most appositely compensates for any image shake.
It is still a further object of the invention to provide an image shake detecting device which permits processing almost in real time, as a quantity of the image shake can be briefly computed during a vertical blanking period, and also permits reduction in size, weight and cost because of a simple circuit arrangement thereof which in principle obviates the necessity of having an A/D converter and storage and computing means on a large scale.
To attain the above-stated object, an image shake detecting device which is arranged as a preferred embodiment of the invention to detect a shake of an image on an image sensing plane on the basis of an image signal output from image sensing means comprises: extracting means for extracting a feature point from within the image sensing plane; detection means for detecting a difference in generation timing of the feature point between a plurality of temporally different images; computing means for computing a quantity of the shake of the image on the basis of an output of the detection means; and sensitivity control means for varying a sensitivity of the detection means.
It is an object of the invention to provide a control device for an image sensing optical system which is very stable and has a good response characteristic.
It is another object of the invention to provide a control device for an image sensing optical system which is capable of performing an image-stabilizing and object-tracing action to effectively compensate for any image displacement resulting from a movement of an object to be photographed and is advantageously applicable not only to a photo-taking optical system but also for image processing systems of varied kinds, on account of the following arrangement thereof: In view of a sampling period for dispersively detected information and the time delay of the detected information existing in cases where the controlled system is the image sensing system of a TV camera or the like, the amount of control operation for a feedback control system is computed and supplied to the controlled system. This greatly stabilizes the control output and improves the response and frequency characteristics. With the controlled system stabilized in this manner, the setting range of feedback coefficients can be broadened for the system.
It is an object of the invention to provide a compact control device which requires no special sensor nor any special optical part.
To attain that object, a control device for an image sensing optical system which is arranged as a preferred embodiment of the invention comprises: first control means for feedback-controlling the image sensing optical system on the basis of detection information obtained by timewise dispersively detecting a state of the image sensing optical system; memory means for storing control information obtained on the basis of a quantity of control performed over the image sensing optical system when the detection information is sampled; and second control means for controlling the image sensing optical system on the basis of a result obtained by computing the control information stored in the memory means and the detection information.
Further, another control device for an image sensing optical system according to this invention comprises: scanning means for scanning at the same speed a plurality of images obtained at intervals of a predetermined period of time; detection means for detecting a time difference between the plurality of images in timing at which corresponding points of feature are generated during a scanning process of the scanning means, by counting the number of clock pulses; computing means for detecting a quantity of a shake of an image by computing an output of the detection means; and sensitivity control means for varying a detection sensitivity of the detection means by changing a period of the clock pulses.
It is an object of the invention to provide an image processing device which is capable of accurately determining an object tracing area by statistically processing movement vectors obtained at a plurality of parts of an image plane.
It is another object of the invention to provide an image processing device which determines an object tracing area so that an object desired by the photographer to be stably positioned within an image plane can be automatically discriminated from other objects and is arranged to accurately operates even in cases where there is no distinct difference in luminance between these objects or where there are a plurality of moving objects.
To attain the above-stated object, an image processing device arranged according to this invention as a preferred embodiment thereof comprises: movement vector detecting means for detecting a movement vector of an image in each of detection blocks formed by dividing an image sensing plane; computing means for detecting a distributed state of an image movement within the image sensing plane by statistically processing the movement vector obtained from each of the detection blocks; and area discriminating means for detecting an area having an image of a main photographed object on the basis of an output of the computing means.
It is an object of the invention to provide an automatic focusing device which is capable of controlling a focus adjusting device on the basis of an optical flow of a plurality of movement vectors on an image plane.
It is another object of the invention to provide an automatic focusing device which is arranged to obtain the optical flow of an image plane by detecting movement vectors obtained within a plurality of blocks set on an image sensing plane, to detect a movement of a photographed object on the basis of displacement of the optical flow, to accurately determine whether the object is approaching to or moving away from a lens and to decide the restart of an automatic focusing action on the lens according to information on the result of determination.
To attain the above-stated object, an automatic focusing device arranged as a preferred embodiment of the invention comprises: focus detecting means for detecting a degree of focusing on an object in an image sensing plane to make focus adjustment; movement vector detecting means for detecting a movement vector of an image in each of detection blocks formed by dividing the image sensing plane; area detecting means for detecting an area having an image of the object by computing the movement vectors in the detection blocks; and control means for detecting a movement quantity of the image of the object on the basis of the movement vectors in the focus detection area detected by the area detecting means and for controlling the focus detecting means on the basis of the movement quantity.
Other objects and features of the invention will become apparent from the following detailed description of embodiments thereof taken in connection with the accompanying drawings.